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matrices. The results on the existence of capacity achieving linear codes in the literature are limited to the random 
binary codes with equal probability generating matrix elements and sparse parity-check matrices. Moreover, the 
codes with sparse generating matrices reported in the literature are not proved to be capacity achieving. 

As opposed to the existing results in the literature, which are based on optimal maximum a posteriori decoders, 
the proposed approach is based on a different decoder and consequently is suboptimal. We also demonstrate 
an interesting trade-off between the sparsity of the generating matrix and the error exponent (a constant which 
determines how exponentially fast the probability of error decays as block length tends to infinity). An interesting 
observation is that for small block sizes, less sparse generating matrices have better performances while for large 
blok sizes, the performance of the random generating matrices become independent of the sparsity. Moreover, we 
prove the existence of capacity achieving linear codes with a given (arbitrarily low) density of ones on rows of the 
generating matrix. In addition to proving the existence of capacity achieving sparse codes, an important conclusion 
of our paper is that for a sufficiently large code length, no search is necessary in practice to find a deterministic 
matrix by proving that any arbitrarily selected sequence of sparse generating matrices is capacity achieving with 



high probability. The focus in this paper is on the binary symmetric and binary erasure channels. 

I. Introduction 

The Shannon coding theorem [OQ states that for a variety of channels with a given capacity C, if the 
information transmission rate R over the channel is below C, there exists a coding scheme for which the 
information can be transmitted with an arbitrarily low probability of error. For Discrete Memory less 
Channels (DMC), it has been shown |[2]| that the probability of error can be bounded between two 
exponentially decaying functions of the codeword block length, n. In this theorem, there is no constraint 
on the codes in terms of linearity. In [|3), a simpler proof of the Shannon theorem has been provided. The 
existence of capacity achieving linear codes over the Binary Symmetric Channel (BSC) was shown by 
Elias [5 1 where it was also proved that linear codes have the same error exponent as random codes. A 
similar result has been obtained in [|6). It was recently shown in that the error exponent of a typical 
random linear code can, in fact, be larger than a typical random code, implying a faster decaying of error as 
n increases. Some bounds on the decoding error probability of linear codes have been derived in [8|. The 
result reported in [E)-[|8] are all based on the fact that the elements of generating matrices of the capacity 
achieving linear codes should be one or zero with equal probability; therefore the generating matrix of 
such approaches are not sparse]!! Moreover, most papers on capacity achieving sparse linear codes are 
concentrated on codes with sparse parity-check matrices. In particular, an important class of codes called 
Low-Density Parity-Check (LDPC) codes [|9]|, IflOll have been of major interest in the past decade. While 
these codes have sparse parity-check matrices, they do not necessarily exhibit sparse generating matrices 
which are the focus of this paper. In |[Tl"l - lfl2ll . some Low-Density Generating-Matrix (LDGM) schemes 
have been proposed which have performance approaching the capacityo Some other related literature on 

'A sparse generating matrix is a matrix with a statistically low density of ones, see Section II for the exact definition. 

2 We distinguish between "capacity approaching" and "capacity achieving" codes. The former term is used when the performance of the 

code can be shown numerically to approach capacity without any guarantee to achieve it. The latter term is used if the performance can be 

rigorously proved to achieve the capacity. The subject of this paper is on the latter case. 



the codes with sparse generating matrices having performance close to capacity includes lfT3l - [fT51 ; in 
lfT3l . a capacity-achieving scheme has been proposed based on serially concatenated codes with an outer 
LDPC code and an inner LDGM code. However, the generating matrix corresponding to the concatenation 
is not necessarily sparse. On the other hand, rateless codes have been proposed in [14] and lfT5l which 
have sparse generating matrices but are only proved to be capacity achieving over the Binary Erasure 
Channel (BEC). 

In this paper, using a novel approach, we prove the existence of capacity achieving linear codes with 
sparse generating matrices that can provide reliable communications over two important classes of DMC 
channels; namely, BEC and BSC at rates below the channel capacity. The proof is accomplished by first 
deriving a lower bound on the probability of correct detection for a given generating matrix and then by 
taking the expectation of that lower bound over all possible generating matrices with elements 1 and 
with probability p and 1 — p, respectively. By showing that this expectation goes to one as n approaches 
infinity, we prove the existence of linear capacity achieving codes. To show the sparsity, we extend this 
result by taking the expectation over a subset of matrices for which the density of ones could be made 
arbitrarily close to any target p. We then prove a stronger result that indicates the existence of capacity 
achieving linear codes with the same low density of ones in each row of the generating matrix. In addition 
to proving the existence of capacity achieving sparse codes, we also show that for a sufficiently large code 
length, no search is necessary in practice to find the desired deterministic matrix. This means that any 
randomly chosen code can have the desired error correcting property with high probability. This is done by 
proving that the error probability of a sequence of codes, corresponding to a randomly selected sequence 
of sparse generating matrices tends to zero as n approaches infinity, in probability. This important result 
is then extended to generating matrices with low density rows for the case of BSC. 

Although in reality the bloclength of codes is finite, in order to prove that a class of codes is capacity 
achieving, we assume that the blocklength goes to infinity. An intersting question is that for a given error 



probability and blocklength, how close the rate of the code can be to the capacity. An upper bound for 
the channel coding rate achievable at a given blocklength and error probability is derived in 0]. In our 
paper we use Yuri's upper bound [4] and other well-known results to compare to our numerically derived 
results. 

An interesting trade-off between the sparsity of the generating matrix and the error exponent is demon- 
strated such that the sparser the matrix, the smaller the error exponent becomes. It is important to note that 
for the case of BSC, we rigorously prove the existence of capacity achieving linear codes for a constant 
p resulting in a non-vanishing density of ones on the generating matrix as n tends to infinity. However, 
we have made a conjecture that if we choose p(n) = 1/n 7 ; where < 7 < 1, the resulting codes can still 
be capacity achieving, which implies a vanishing density of ones. This signifies that the number of ones 
in the generating matrix can be as low as n 2 ^ 1 . For the case of BEC, we have been able to prove that 
to have capacity achieving generating matrices, p{n) can be of O(^p). This implies that the number of 
ones in the generating matrix is about nlogn which is asymptotically less than n 2 ~' 1 , the number of ones 
in the case of BSC. As opposed to the existing results in the literature, which are based on Maximum A 
Posteriori (MAP) decoders, the proposed proofs are based on a suboptimal decoder]^ which makes our 
approach also novel from decoder point of view. 

The organization of the paper is as follows: In the next section, some preliminary definitions and 
notations are presented. In Sections [III] and [TV] we present our theorems for BSC and BEC, respectively, 
and Section [V] concludes the paper. 

II. Preliminaries 

Consider a DMC which is characterized by X and y as its input and output alphabet sets, respectively, 
and the transition probability function ¥{y\x), where x E X is the input, and y E y is the output of the 
channel. In this paper, we consider the binary case where X = {0, 1}. A binary code C(n, k) of rate R is a 

3 See the details in the next section. 



mapping from the set of 2 k fc-tuples X { to n-tuples Z u < i < 2 k - 1, where X { e {0, l} k , Z { e {0, 1}™, 
and the code rate R is defined as the ratio of k by n. Since we are only interested in Linear Codes, the 
mapping is fully specified by an n x k binary matrix A = {Aij} (the generating matrix), and encoding 
is accomplished by a left multiplication by A: 

Zi = A-Xi, 

where the calculations are in GF(2). The vector Zi is then transmitted through the DMC. Decoding is 
defined as recovering the vector Xi from the possibly corrupted received version of Z^ 

In this paper the employed decoding scheme relies on the a posteriori probability distribution. Let A 
be the generating matrix. For a received vector Y — y, the decoder allocates a random vector such as 
X = x as the original transmitted message with the conditional probability F(X = x\Y = y). Clearly, 
the probability of correct detection using A as the generating matrix is 

p c (A) = Y i V(x i )F(y j \x i )V(x i \y j ) = ^P(^, % -)P(^| % -) = E X>Y {¥{X\Y)), (1) 

where F(X, Y) depends on A. 

Note that the optimal decoder is a MAP decoder which allocates argmax x ¥(X = x\Y = y) and that 
the probability of correct detection using MAP is more than or equal to the probability of correct detection 
in (OQ). Throughout the paper, the index i in X t and Z,i may be dropped for more clarity. For the sake of 
convenience, the following notations are used for the remainder of the paper. 

Definition 1: Let A n xk be the set of all binary n x k matrices. The density of an A G Anxk is defined 
as the total number of ones within the matrix divided by the number of its elements (nk). A matrix with 
a density less than 0.5 is called sparse; the smaller the density, the sparser the matrix becomes. 

Definition 2: Let each entry of each element of A nX k has a Bernoulli(p) distribution, < p < lo This 
scheme induces a probability distribution on the set Anxk, denoted by Bernoulli(n, k,p). For the rest of 
paper, we consider this distribution on the set A n xk- 

4 A binary random variable has Bernoulli(p) distribution if it is equal to 1 with probability of p and equal to with probability of 1 — p. 



Note that as n approaches infinity, the typical matrices of A nX k have a density close to p. 

III. Binary Symmetric Channel (BSC) 

Consider a BSC with cross-over probability e. The capacity of this channel is given by C = 1 — h(e), 
where h (e) = — eloge — (1 — e) log (1 — e). We suppose that R, the rate of the code, is less than C. In 
this section, we prove the existence of capacity achieving linear codes with arbitrarily sparse generating 
matrices over the BSC. We prove the existence by showing that the average error probability over such 
generating matrices tends to zero as n approaches infinity. 

A. Channel Modeling 

Assume that we encode a message vector X to generate the codeword AX. Note that X is chosen 
uniformly from the set {0, l} k . Due to the effect of error in the BSC, each entry of the transmitted 
codeword AX can be changed from to 1 and vice versa. These changes can be modeled by adding 1 to 
erroneous entries of AX (in GF(2)). Therefore, the error of a BSC with cross-over probability e can be 
modeled by a binary n-dimensional error vector N with i.i.d. entries with Bernoulli(e) distribution. Thus, 
if the output of the channel is shown by Y, the following equation models the channel: 

Ynxi — A nX fcXfc X i + N nx i. (2) 

Note that X and N are independent. 

B. Capacity achieving sparse linear codes for the BSC 

In the following theorem, a lower bound for the average probability of correct detection over the set 
Anxk, is obtained. 

Theorem 1: Consider a BSC with cross-over probability e. A lower bound for the average probability 
of correct detection over all n x k generating matrices with Bernoulli (n, k, p) distribution is given by 

^2(n-i) 

EL, ©(I " (1 - 2c)(l - 2p)i)'(l + (l-2e)(l-2p)0»- i ' 



®AeA xk (Pc (A)) > y ( U ) x . 2W ^ (1 eY ' . (3) 
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Fig. 1. A comparison of the coding rate versus the blocklength for various values of density(p), e = 0.11, Capacity=0.5, error 
probability=10 _1 . 

Proof: See Appendix II. 

Note 1: An important result of this theorem is that we can fix the error probability and find the maximal 
achievable rate for a given blocklength. See the following figures. 

Figure 1 is a plot of the coding rate versus n for p equal to 0.1, 0.3 and 0.5. This plot is numerically 
evaluated from Theorem CD An interesting observation of this figure is that when the blocklength n 
increases, the coding rate becomes independent of the density p. This observation can be shown to be 
true from (1221 of Lemma |2] where the parameter p disappears on the right hand side. The significance 
of this observation is that sparse generating matrices can replace non-sparse ones for large block coding 
sizes, which implies simpler encoder design. This observation is the dual of LDPC codes where large 
sparse parity check matrices simplifies the decoder design, while the performance remains the same. 

Figure 2 is a comparison of our result to that of Gallager result and Yuri upper bound [4J. This figure 
shows that our results are within the Yuri upper bound and the Gallager result. This figure also shows 
that for the probability of error equal to 10~ 3 when n becomes greater than 180, the performance of the 
sparse genearting matrices with p = 0.3 becomes the same as the non-sparse matrices with p = 0.5. 
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Fig. 2. A comparison of the coding rate versus the blocklength for various methods; the Gallager and the Yuri curves are plotted from (4), 
e = 0.11, Capacity=0.5, error probability=10 -3 . 

In the following theorem, we will show that the expected value of the correct detection probability over 
all generating matrices from A n xk approaches 1. This proves the existence of at least one linear capacity 
achieving code. 

Theorem 2: For any < p < 1, for a BSC we have 

lim E AeAixfc (p c (A)) = 1. (4) 

n— >oo 

Proof: See Lemmas 1 and 2 and the proof in Appendix III. 

The performance of linear codes is determined by the error exponent which is defined as follows: 
Definition 3: The error exponent of a family of codes C of rate R is defined as 

E C {R) = lim logp e , (5) 

n— >oo n 

where p e is the average probability of decoding error. 

If the limit is greater than zero, the average error probability of the proposed codes decreases expo- 
nentially to zero as n increases. The error exponent is an index such that the larger the error exponent, 
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Fig. 3. The average error probability versus n, e = 0.05, R = 0.8C 

the faster the probability of error decays as n increases. Based on our observation, there is an interesting 
relation between the error exponent of the codes constructed by generating matrices with Bernoulli(ra, k, p) 
distribution and the values of p. In Fig. 3, we have plotted the average probability of error versus n for 
various values of p. As it can be seen, the error exponent which is equal to the slope of the curves, increases 
as p increases (the generating matrix become less sparse). In other words, although the probability of error 
for sparse codes goes to to zero exponentially as n increases; this decrease is not as fast as high density 
codes. 

Definition 4: Let W(A) be the number of ones in a given binary matrix A and r] be an arbitrary 
positive constant. T^ xk is defined as a subset of A nX k for which | W k — p\ < rj, r] > 0. By choosing a 
sufficiently small t], the set T% xk is in fact a subset of A nX k which contains matrices having density of 
ones arbitrarily close to any given p. Note that the probability distribution on T% xk is induced from the 
probability distribution on A nX k- 

In Theorems \T\ and [21 we proved the existence of capacity achieving codes for any value of p. We 
did not explicitly prove the existence of sparse capacity achieving codes. However, using concentration 



theory |fT6l , we can see that for a sufficiently large n, a randomly chosen matrix from A nX k is in the 
subset T^ xh with high probability. In other words, we can state the following proposition which implies 
the existence of capacity achieving codes which are sparse. 

Proposition 1: Let T^ xk be the set of typical matrices defined in Definition ©. We then have 

lim E Aen he) = 0. (6) 

Definition 5: We define 1Z nX k as the set of all binary nxk matrices with rows that have hp ones. We 
also consider a uniform distribution on the set TZ nX k for the rest of the paper. 

In the next theorem, we will prove a stronger result on capacity achieving sparse codes. We show the 
existence of capacity achieving matrices with rows containing exactly hp ones. In other words, the density 
of ones in each row is exactly equal to p. This also implies that the generating matrix has a density of 
ones exactly equal to p. In Theorem |3j we shall derive a lower bound on the average probability of correct 
detection and in Theorem @] we will prove that this lower bound tends to one. This shows that the average 
probability of error over the set lZ nX k approaches zero, implying the existence of capacity achieving codes 
with generating matrices taken from !Z nX k- 

Theorem 3: For a binary symmetric channel with cross-over probability e, a lower bound for the 
expected value of the probability of correct detection over all generating matrices in 1Z nX k is given by 

Ea^ x . (Pc (A)) > £ r W(l - e)^ £ [ - (7) 

i= o W Ej=o U ( eA J + WAV 1 - e ) A 3 + eB j) 

where 

Ai = a E i> 6) - J ■ B ' = , L i 6) w - J J ■ 

Proof: See Appendix IV. 

Theorem 4: For each < p < 1, we have 

limE Ae7W (p c (A)) = l. (8) 



Proof: See Lemma 3 and the proof in Appendix IV. 

In Theorems [TJ and |2l we proved the existence of capacity achieving linear codes with generating 
matrices having Bernoulli^, k, p) distribution by showing that the average probability of error over all 
generating matrices tends to zero as n approaches infinity. This implies that we may have to perform 
a search over A nX k to find such a matrix. Assume that we simply pick matrices randomly for each n 
from the set A nX k- This constitutes a sequence of n x nR matrices. Now consider the resulting sequence 
of error probabilities corresponding to the sequence of generating matrices. In the following proposition, 
we shall prove that the limit of this sequence is zero in probability, i.e., a sequence of randomly chosen 
matrices is capacity achieving with high probability. This suggests that for sufficiently large n, no search 
is necessary to find a desired deterministic generating matrix. 

Proposition 2: Let {A nxn #}^L be the sequence of matrices, where A nxn R is selected randomly from 
A n xnR- If we denote the error probability of the generating matrix A nxn ^j over BSC by p e (A n ), then 
p e (A n ) converges in probability to zero as n tends to infinity. 
Proof: See Appendix V. 

Note 2: If we use the result of Theorem @] we can extend Proposition |2] to the case where we construct 
the matrix sequence by choosing the matrices from the set 1Z nX k- hi other words, in order to have capacity 
achieving sequences of generating matricescfor BSC with arbitrarily low density rows, we can simply pick 
generating matrices randomly from TZ nX k- 

At this stage, we have been able to rigorously prove the existence of capacity achieving sparse linear 
codes over the BSC. However for a given p, although the density of ones can be made arbitrarily small, 
it does not go to zero even when n approaches infinity. Let us assume the case where p is a decreasing 
function of n such that lim^oo p{n) = 0, resulting in zero density of ones as n goes to infinity. In the 
following conjecture, we will propose a result indicating that this assumption can in fact be true. Although, 
we have not been able to rigorously prove the conjecture, a sketch of the proof has been presented in the 



appendix. 

Conjecture 1: Let 7 be an arbitrary number from interval (0,1). For p(n) = ^ by assuming the 
Bernoulli(n, k,p(n)) distribution on the set A nX k, we have 

limE AeAnXfc (p c (A)) = l (9) 

n->oo 

See Appendix V for the sketch of the proof. 

IV. Binary Erasure Channel 

A binary erasure channel is identified by erasure probability e and the capacity of this channel is given 
by 1 — e. We use the decoder proposed in Section II. Through the channel, some entries of the coded vector 
AX, shown by Z, may be erased. According to the position of the erased entries, the error of the channel 
can be modeled as a subset F of F = {1, . . . , n}. Therefore, we employ a decoder which decides about the 
transmitted vector by observing only the non-erased entries denoted by Z F . For each i e F, the i th row of 
A is removed to derive A F . Therefore, the encoding and channel operation can be written as Z F = A F X. 
The decoder chooses X, the estimation of X, randomly from the set X{Z,F) = {X\A F X = Z F }. 
In this case, the decoder is equivalent to the MAP decoder. From linear algebra, it can be shown that 
\X(Z,F)\ = 2 k ~ rank ^ AF \ where rank is the maximum number of independent rows of a matrix calculated 
in GF(2). Since X is chosen uniformly from X(Z, F), the probability of the correct detection of X is equal 
to 2-( fe - ranfc ( A ^)). Thus, we have P c \x,f( A ) = F ( X = x \ X i F ) = 2 rank ^~ k , where p c]x , F represents 
the probability of correct detection when X is transmitted and the position of erased entries are given in 
F. 

Theorem 5: Let C be the capacity of a BEC and A e A nX k is a generating matrix corresponding 
to a code of rate R < C. For any p{n) of O(^p), the expected value of p c (A) over all matrices with 
Bernoulli(n, k, p(n)) distribution tends to 1 as n approaches infinity. 

lim E AeAix > c (A)) = 1. (10) 
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Proof: See Appendix VI. 

From the concentration theory [fl6l . similar to the case of the BSC, we can state the following 
proposition. 

Proposition 3: For a BEC with capacity C, codes of rate R < C and generating matrix from T^ xk , 
we have: 



In the following proposition we show that similar to Proposition [2] for BSC, a sequence of randomly 
chosen generating matrices from A n xnR, results in a capacity achieving coding scheme with high prob- 
ability. This suggests that for sufficiently large n, no search is necessary to find a desired deterministic 
generating matrix. 

Proposition 4: Let {A nxnfl }~ =0 be the sequence of matrices, where A nxnR is selected randomly from 
A nxnR- If we denote the error probability of the generating matrix A nxn ^j over BEC by p e (A n ), then 
p e (A n ) converges in probability to zero as n tends to infinity. 
Proof: The proof is similar to the proof of Proposition |2] and thus omitted. 



lim E Aer: (p e (A)) = 0. 



(11) 



In Fig. 2, we have shown the error exponent as a function of j£ for different values of p. As it can be 
seen, a similar trade-off to BSC exists between sparsity and the error exponent. The smaller p results in 
a smaller error exponent. 

The following theorem is similar to Theorem |4] 

Theorem 6: For each < p < 1, for a BEC we have 

KmE A6Wmxfc (p e (A)) = l. (12) 

n— >oo 

Proof: See Appendix VII. 

Note 3: If we use the result of Theorem |6l we can extend Proposition |4] to the case where we construct 
the matrix sequence by choosing the matrices from the set 1Z nxk . 

V. Conclusions 

In this paper, a novel approach to prove the existence of capacity achieving sparse linear codes over 
the BSC and BEC was proposed. For the BSC, in Theorem \T\ we derived a lower bound on the average 
probability of correct detection over the set A nX k- In Theorem [21 we proved that the average probability 
of error over A nX k tends to zero. Then we proved the existence of sparse capacity achieving codes in 
Proposition 2. In Theorem |3j we derived a lower bound on the average probability of correct detection 
over the set lZ nX k- Using this lower bound in Theorem @] we proved the existence of capacity achieving 
codes with generating matrices with the same density (p) in each row. In Proposition |2] and its preceding 
note, we showed that the error probability of codes corresponding to any randomly chosen sequence of 
generating matrices tends to zero in probability. This implies that for sufficiently large n, a randomly 
chosen matrix from A nx t and lZ nX k will have the average error correcting capability. In addition, we 
conjectured that Theorem |2] can hold for the case where p is a function of n, i.e. p = l/n 1 . This implies 
that for a capacity achieving code over a BSC, the density of the generating matrix can approach zero. 
In Theorem \5\ and Proposition 3, we proved the existence of sparse codes for the case of BEC with 



generating matrices having Bernoulli distribution with p(n) of 0(-^ !1 ). A similar result to Proposition |2] 
and Theorem @] was shown for BEC in Proposition @] and Theorem |6] respectively. We demonstrated an 
interesting trade-off between the sparsity of the generating matrix and the error exponent indicating that 
a sparser generating matrix results in a smaller error exponent. We also observed that for small block 
sizes, generating matrices with higher densities have better performances while for large block sizes, the 
performance of the random generating matrices become independent of the density. In our proofs, we 
have used a suboptimal decoder while previous works in the literature were based on a MAP decoder. 
This implies that we can get stronger results if we use the optimal MAP decoder. 

For future work, one can try to rigorously prove Conjecture 1 and possibly extend it to the case of 
matrices in the set lZ nX k- The improvement in the bounds using a MAP decoder can be an interesting 
topic to investigate. The extension of the results to DMC's is another challenging topic to be explored. 
A very interesting work is to analytically derive the error exponent to prove the trade-off between error 
exponent and sparsity of the generating matrix. 



Appendix A 



We need the following definitions in order to prove our theorems. 

Definition 6: Any two functions a(n) and b(n) are referred to as proportionally equivalent and written 
as a(n) « b(n) if lim^^ ^ = 1. 

Definition 7: Any two functions c(n) and d(n) are referred to as differentially equivalent and written 
as c(n) = d(n) if lim^oo c(n) — d(n) = 0. 

Appendix B 
The Proof of Theorem [J 

Proof of Theorem |7J According to CQ), Bayes' rule, and the independency of X and N, we have 

Pc(A) = E x , y ^ p(y) ) = E X)N { p(y) J • d3) 
Taking expectation over all matrices A G A nX k, we get 

«Ue,„Um«. (- 1 



E A (p c (A)) = E A , X , K I ' ' I = Ex, K I P(iV)P(X)E A I p?py ] I , (W) 

where in the last equality, the independency among A, iV and X is used. Using the Jensen's inequality 
(see IfTTl, Chapter 2, Page 25), we have 



E A (p c (A)) >E XiJV 



P(iV)P(X)\ 

E A (F(y)) ; 



P(iV)P(X) 



E A (P X ^(AX + N = AX' + jV')) 

¥(N)¥(X) 
E A (Ex',jv(l[ A (x-x')+(iv-JV')=o])) 



, , P(AQP(X) \ 

1 E x ,(F AiN ,(A(X-X') + (N-N>) = 0)))> 1 ; 

where X' and A r ' have the same distributions as the input and error vectors, respectively. In the above 

equation, the expected value over X' is a function of binary subtraction X — X' and as a result does 



not depend on X. Thus we can assume any binary vector X such as the all zero vector, X ; from the 
independency of the rows of A in (fT5l) and the uniformity of the vectors X, we have 



if F(N) 
E A «A)) >^E K , M ( Ejf , (nr _ iPjliiJ<!(yllW _ x , ) + (JVl _ w?) = 0)) I < 16 > 



2" \E X , (nit V A „ N; (A(X') + (AT, - Nfi = 0)) 
where N t and AT/ are the I th entry of A" and N', respectively, and Ai is the I th row of A. Note that here 
all the operations are performed in GF(2). In order to evaluate the right side of the above inequality, 
assume that vector A" has i ones. Without loss of generality and for convenience, we assume that the first 
% elements of A^ are 1. Thus, the argument of the expected value in ([TBI is equal to 

—T-. v (17) 

Ex- (rc,i ( F i,«; w + N i = !)) nr., + i (p*,«r w + «? = o))j 

To evaluate the expected value in the above expression, note that since N[ = with probability 1 — e 
and N[ = 1 with probability e, we have 

i 

11 (F Al>N , (AX' + N[ = 1)) = (e x P (AX' = 0) + (1 - e) X P (AX' = l)) 1 . (18) 
i=i 

Now assume j elements of X' are equal to 1. Also consider the entries of Ai with the same indices as 
the entries of X' that are equal to one. It is easy to see that in the above equation, P (A\X' = 1) is equal 
to the probability of having an odd number of ones in the considered indices of A\. Thus, we have 

W = i) = £ ( J >0 - = '^-^l'' 1 -^ ) = l^lEl. (19) 

q odd 

The same argument results in P (AiX' = 0) = l±ilzjgH an( j therefore we have 



n P ^, {MXo _ xo + w - m) = o) = ( i -' i - 2 f - 2 "' j y ( i+(i - 2 f-^ ) 



n—i 



(20) 



The expectation of the above expression over X' results in 

/ n 



£ 1 Q ^ l-(l-26)(l-2p)^ 2 ^ l + (l-2e)(l-2p)^ n " 1 _ (21) 

Substituting (l2~TI) in ([171) and taking expected value with respect to N, we obtain the following lower 
bound for E A (p c ): 



i=0 x 



.n—i 



E-=o(l " (1 " 2e)(l - 2p)i)*(l + (1 - 2e)(l - 2p)i)- 



This completes the proof. 



Appendix C 
Lemmas [B[2] and the proof of Theorem [2] 

Lemma 1: Let {cii}°^ be a bounded sequence. For any 5 > and < p < 1 the summation 



ELo O^l -P) n-i a* is differentially equivalent to ES p -« 0^(1 ~P) n 



i=n(j>— 5) 



di. 



Proof: According to the Chernoff-Hoeffding Theorem [19J the proof is straightforward 



Lemma 2: Consider a code with rate R over a BSC with cross-over probability of e where R = k/n < 
C = 1 — h(e). There exists a <5 > for which for any i £ (n(e — 5), n(e + 5)), we have 



£ (*) (1 - (1 - 2e)(l - 2p)0*(l + (1 - 2e)(l - 2^ 



yy-i ~2V(l-e)™- 1 . (22) 

i=o v ' 7 

Proof: To prove the lemma, note that the first term of the summation (j = 0) in the left hand side of (l22l) 



is equal to the right hand side. Therefore, to prove (1221 . it is sufficient to show that 



Let b(j) = M^MOzg^ and a (^) = i-d-fcKi^y - i+a-jeXi^ ) " ) . Thus, we have 



i 



l-e 

By using a straightforward calculation, it can be shown that for % = ne, the maximum of a(j, i) is equal 
to 1. The maximum of a(j,i) occurs for b(j) = e or equivalently j = 0. Thus, for j > 1 we have 



a(j>e)<l. (25) 



Also, since lim 3 _ ) . C)0 (6(j)) = |, we have 



i 



lim a(j,ne) = I ( ^- ) (7777- — 7 ] = < I. (26) 



It is easy to see that a(j,i) is a uniformly continuous function of i and j. Thus from (|25l . we conclude 
that there is a 5i > for which for any i E (n(e — 8i),n(e + §1)) and j > 1, we have a(j,i) < 1. And 
also from (|26l ), we conclude that there is a <5 2 > for which for any i E (n(e — 8 2 ), n(e + <5 2 )), we have 
lim^oo a(j, i) < |. Let 5 = min(5 1 , 5 2 ) and fix i E (n(e — 5), n(e + 5)); there exist an integer M and a 
real number // > 0, for which we have a(j, z) < | — // for all j > M. By using this M, the left hand side 
of (|23l ) can be written as 



E(')»^)'=E(')«ai)'+ t < 27 > 

i=l V ^ / / j=l j=M+l 



3=1 3 

Since a(j, i) < \ — jj, for j > M, we have 



t ( •) 0' ^ >™ ( E g) ) (5 - ^ fit 2 '<i - - »■ < 28 > 



Therefore, lim^^ E,=m+i {j) a U^) k = °- 

To see that the first term at the right hand side of (l27b also tends to zero, let w = maxi<j<A/ a(j, i) < 1. 



Therefore, we can write 

M / , N / M 



where v = — \n(w) > 0. Now the right hand side of the above inequality tends to zero because vk — 
Mln(fc) tends to infinity as k approaches infinity. This proves that the left hand side should also tend 
to zero. Therefore, both summations at the right hand side of (|27l) tend to zero. This proves (|23l ) and 
consequently (|22l) . ■ 
Proof of Theorem [2J 

Let a* = =j. — rp- r — ; — - — ~~~~ r — ; - — — r. The first term of the summation of the denominator 

E - =0 (J) (l-(l-2e)(l-2p)i )« (l+(l-2e)(l-2p)J )»- 

is equal to the numerator, and the other terms in the summation are positive. Thus, the elements of the 
sequence {aj}" =0 are less than 1 and subsequently bounded. Therefore, we can apply Lemma 1. Now note 
that based on Theorem \T\ we have 

Eaba .WA^yW^l-rS 2V(l-e) n - t 

X ££W E-=o(l-(l-2 e )(l-2pmi + (l-2 e )(l-2^)- 

Let <5 be as in Lemma 1. Since E, Ae A nxk (p c (A)) < 1, to prove the theorem, it is enough to show that the 
right hand side of the above inequality is differentially equivalent to 1. To see this, we write 



i=o v 7 



£* (1 - (1 - 2e)(l - 2p)i)«(l + (1 - 2e)(l - 2p)i)' 



^ W Eto(! " (1 " 2e)(! " WO + (1 " 2e)(l - 2^)"- 



i=n(e— 5) 



n\ „.,.. 2 n e l (l — e) . sr^ i n 

i 



i=n(e-<5) V 7 ' 



i=0 

where we used Lemma 1 in the first and third equality^! and we replaced the summation in the denominator 
based on Lemma 2. This proves the theorem. ■ 

Appendix D 
The proof of Theorems |3] and S 

Proof of Theorem |3] 

We follow steps similar to that of Theorem \T\ Equations <TT~3T > to (fTTT ) in Theorem \T\ still hold here. It can 

5 Note that by equality we mean = which is not mathematically precise but we use it throughout the paper for the ease of explanation. 



be easily seen that P (AiX' = 1) = A,- and P (A t X' = 0) = Bj, Thus equation CED is modified as 

n 

Hv Ai ,n> (A(X') + (JV, - iV/) = 0) = (eA, + (1 - e)^)' (e^ + (1 - e)A 3 ) n ~ i . (30) 
z=i 

The expectation of the above expression over X' results in 

E x> ( f[^A lt Ni (MX - X') + {Ni - N{) = 0) 

£ ¥ (T) (e ^' + (1 " e) ^ r {eB > + (1 " e) ^ r " 4 • (31) 
3=0 KJ/ 

Substituting (I3TI) into (fT7l) and taking the expectation with respect to N, we obtain 

, sn -i e\l-e) n ~ l 

k /k\ 



E A e^(Pc(A))>^(") 6*(l-6 



e; =0 © + (i - *w + (i - ^r* 

This completes the proof. ■ 
Proof of Theorem |?J 

First we prove a lemma similar to Lemma |2] 

Lemma 3: Suppose that R = k/n < 1 — /i(e). There exists a 5 > for which for any i £ (n(e — 
5), n(e + 5)), we have 

E 00 + o- - e )^) 1 + ^ - e )^) n_i w e ^ - e ) n_i - < 32 > 

i=o ^ 

1 

(i_ n — i \ 

nMIk&iA n Mj+ii-^AA n \ _ Since = ^ 



a{3 , i)={{ ^Y(h^my ] . (33) 



we have 



1-e 

By employing the same approach as the proof of Lemma |2l it is sufficient to show that 

lim b(j) = \. 

j— )-oo Z 

It is easy to see that hin^oo A, = limj^ 00 £>j = ~. As a result, we have 

lim b(j) = lim (eAj + (1 - e)^) = t + lz± = I. (34) 

j— >oo j— >oo Z Z Z 



This completes the proof of lemma. ■ 

Now to prove this theorem, it is enough to replace the denominator in summation of © with the right 
hand side of (|32|) according to Lemma |3] ■ 

Appendix E 

Proof of Proposition [2] and Proof Sketch of Conjecture 3] 
Proof of Proposition |2j 

In order to show that lim n _ s . 00 p e (A nxn / ? ) = in probability, we have to show that for any given 5 > 0, 

lim F(p e (A nxnR ) >6) = 0. 

n— >oo 

For a given £ > 0, define j3 = min{<5, £}. According to Theorem |2] we have lim^co E(p e (A nxn #))) = 0. 
Thus, there exists an Np for which for any n > Np, E,(p e (A nxnR )) < f3 2 . Therefore, due to the fact that 
Pei^nxnR) > 0, for u > Np, we obtain F(p e (A nxnR ) >/?)</?. Hence, for n > Np, since < 5, we 
have 

F{p e {A nxnR ) >5)< F{ Pe {A nxnR ) >P)<P<£. 
Thus, for n > Np, we have 

F{ Pe {A nxnR ) > 5) < £, 

and the proof is complete. ■ 

Sketch of proof of Conjecture |7J 
The lower bound of Theorem \T\ still holds for the case where p is a function of n where < p(ri) < 1. If 
we can show that for R = k/n < 1 — h(e), there exists a 5 > for which for any i e (n(e — 5), n(e + 5)), 
we have 

E (1 - (1 - 2e)(l - 2p(n)y) i (l + (1 - 2c) (1 - 2^^))^)^ « 2V(1 - e)""\ (35) 



From the approach similar to that of the proof of Theorem |2] the proof will be straightforward. Although, 
we have numerical evidence suggesting that the above equality holds, we have not been able to prove it 
rigorously. The rest of the proof is as follows. Let 

2V(l-e) n_< 

^ " £*=o (■)(! " (1 " 2e)(l - 2p(n))i) i (l + (1 - 2e)(l - 2p(n))i)«-' 
Since, the first term of the summation in the denominator is equal to the numerator, the sequence {aj}" =0 

are less than 1 and subsequently bounded. From Lemma \T\ we get 

i=0 ^ ' i=n(e-<5) ^ ' 

Now we have 

E A (p c (A)) > V f n V(l - 2V(l- e y' 



i=0 



In other words, lim^oo E Ae _4 nxfc (p c (A)) = 1 which is the desired result. 

Appendix F 
The Proof of Theorem [5] 

Proof of Theorem |5j 

We first present a lemma from [18]. 



Lemma 4: Suppose 5 > and let Bernoulli(n (1 + 5) , n, p(n)) be the probability distribution on the 
n (1 + 5) x n matrices where p{n) is of O(^p). Then E (rank (A„( 1+( 5) xn )) n. 

Since fc/n < 1 — e, it can be concluded that there exists a <5 > for which k = n(l — e — 5). By 
using the proposed decoding scheme and by decomposing p c |x(A) according to the position of the erased 



entries F, we get 

J9 C | X (A) = P(X = X\X) = ^ P(X = X\X, F)P(F) = ^ e |F| (l - e )*H*1 2 ™nfc(A F )-fc_ 

Therefore, p c |x(A) is the same for all X's. Thus p c (A) = p c \x{A). By evaluating the expected value of 
p c (A) over all matrices and using Jensen inequality, we have 

E A (p c (A)) = ^ ("Vt 1 - e)""E A ( 2 ™^( A («-)x fc )-fc) > ^ W £ <(i _ e )™-* 2 E A(™« fc ( A (-Ox*))-fc. 

i=0 i=0 ^ l ' 

Applying Lemma [H we obtain 

n / \ n{e+9) . . 

I n j e i^i _ e ^ n -*2 E A(™™ fe ( A (n-i)xfc))- fe = ^ / n j e *^x — e ) n ~ i 2 EA ( ranfe(A(n - i)xfe - ) )~ A: 

i=0 i=n(e-0) 

where 9 is chosen such that 9 < 5. For each is (n (e — 6>) , n (e + 9)), there is an a G (— 6 1 , 6>) for which 
i = n (e — a). Therefore, n — i — n(l — e + a) > n(l — e — 8) — k. Now, according to Lemma @] if we 
substitute k for E (rank(A^ n _ i - )><k )) , as n — > oo, we can write 

n(e+9) , . 

E A (p c (A)) > ( J^ 1 - e) n ^2 E ( ranfc(A (— >**>)-* 



i=n{e-e) 
l(e+ 



i=n(e 
n(e+ 



f E(ranfc(A (ra _- )xfc )) 

V 



i=n(e-fl) 7 *=0 

Therefore, lim^ E A (p c (A)) = 1. ■ 

Appendix G 
The proof of Theorem |6] 

Proof of Theorem |5j 

According to the proof of Theorem [5] it is sufficient to show the following lemma. 

Lemma 5: Suppose 5 > and consider lZ n (i-s)xn with its previoiusly defined distribution. Then for 

A n (i_5) xn G 7Z n (i-6)xn we have E (rank (A n (i-5) X n)) ~ ^(1 — £)■ Note that here rank is calculated in 
GF(2). 



Proof: In order to prove the lemma it is sufficient to show that 

lim P (rank (A n n S)xn ) = n(l - 5)) = 1, 

n— >oo 

which is equivalent to show that the probability of having a matrix A n (i_5) xn with linear dependent rows 
goes to zero as n approaches inifinity, i.e., 

Sz 5) ) r{Ai + M + • + Ak = 0) ) = °- <37) 



k=l 

■th 



where A; represents the r row of the matrix. Suppose ( be a positive number such that ( < p. The 



summation of equation (1371) can be written as 

£ ( n(1 ~ 6) )nAi +A 2 + ... + A k = 0) + ( n{1 ~ 5) W + A 2 + ... + A k = 0). (38) 

k=l ^ 7 k=nC, ^ 7 

We first prove that the first term tends to zero. In order to have A\ + A 2 + . . . + A k = 0, A k should be equal 
to the sum of A\ to A k _i. Thus, conditioning on Ai to A^-x, it is easy to see that F(A 1 + A 2 + ... + A k = 



0)<j±t. Thus, 

\pn) 



k=l v 7 \fc=l V 7 / Vpr; 



) 



Since « we have 



f V s f 71 ^ 1 r n(2 nh ^2~ nh ^ 



where in the last equality we used the fact that < /i(p). 

in order to complete the proof it is sufficient to show that the second term of equation (1381 ) also goes 
to zero. In this regard we show that for sufficiently large n and any k > (n we have 

F(A 1 + A 2 + ...+A k = 0) < C^zi) • (39) 
This will prove the lemma because we would have 

n(l-<5) 



lim V ( n{ \ 6) )F(A 1 + A 2 + ... + A k = 0)< 

k=n( v ' 



k=nC, 



k J / V2 1_ 2 / n->o° V2 1_ 2 



In order to prove equation (|39l ) we employ coupling method from random walk theory. Consider 
a random walk on the n-dimensional cubic in GF(2) with the set of directions S, consists of all n- 
dimentional vectors with pn ones. Suppose this random walk starts from the origin, and each time selects 
its next direction randomly from S with uniform distribution. Therefore, ¥(Ax + A 2 + ... + A k = 0) 
represents the probability of returning back to the origin after k steps. Denote this random walk by the 
sequence {X t } of n-dimentional vectors where X t represents the position of the random walk after t 
steps. Note that the stationary distribution of this random walk is uniform distribition, which means as t 
tends to infinity the probability of being at any points of the cubic is almost (|)"- Thus, for large values 
of k, F(Ai + A 2 + ... + A k = 0) is almost (|) n . Now Consider another random walk denoted by {Y t }, 
which its starting point is selected randomly with the uniform distribution. The idea of coupling is to 
couple two random walks {X t } and {Y t } with the dependency between the directions selected by them 
such that both of them remain random walks that select their directions in each step uniformly form S. 
Suppose Xi and Yj are the positions of the two random walks after i steps and sf +1 be the (i + l) th 
direction which is selected uniformly from S by the random walk {X t }. Suppose rj entries of the vectors 
Xi and Yi are the same and denote the positions of these entries by the set U = {ui, U2, u n }. Let 
Su be the subset of S consists of vectors that their r, entries with positions from U are same as sf +1 . 
The random walk {Y t } select the direction s| fl uniformly from the set Su. Note that due to the fact 
that {Y t } starts with its stationary distribution the probability of being at any point remains uniform for 
all t for this random walk. Also note that according to the dependency between s* +1 and sf +1 , {rj} is a 
non-decreasing sequence. Thus, we expect that the two random walk meet each other at a point. Let r 
be the first time that {X t } and {Y t } meet. Note that after r the rest of the two random walks would be 
the same. Conditioning on the r, P(X^ = 0) can be written as 

p(X fc = 0) = P(X fc = 0|r < fc)P(r < k) + F(X k = 0|r > fc)P(r > k). 



Now if we can prove that for k > (n, P(r > k) goes to zeros as n tends to infinity, then we would have 

lim P(X fc = 0) = lim F(X k = 0|r < jfe)P(r < k) + F(X k = 0|r > fc)P(r > k) 
= lim F(Y k = 0|r < fc)P(r < fc) + lim P(X fc = 0|r > fc)P(r > fc) 

n— >oo n— >-oo 

= lim (i) n P(r < k), 

which proves the equation (|39T ). Note that P(r < fc) tends to 1 as k approaches infinity. Therefore to 
complete the proof it remains to show that 

lim P(r > Cn) = 0. (40) 

n— >oo 

For 1 < j < n, suppose Tj represents the first time that the j th entries of X t and Y t become the same. 
Thus we have 

n 

P(r > Qn) < ^P(r j > (n) = nP(n > (n). 

Suppose that after % steps the first entries of Xj and are not the same and let r, and the set U be as 
defined previously. Let p <\. The first entry of sf +1 is equal to one with probability p. Now due to the 
fact that p <\, less that entries of sf +1 which are not from U are equal to one with a probability 
more than |. This means that the first enrty of s v i+l is equal to zero with a probability more than |. Thus, 
the first entries of and sf +1 are not the same with a probability more than |. A similar approach for 
the case p > ~ shows that there is a positive probability p independent from n and i such that the first 
entries of sf +1 and sf +1 differ, i.e., the first entries of and are the same. Thus we have 

lim P(r > Cn) < lim nP(ri > Cn) < lim n(l - p) n ^ = 0. 

n— >oo n—^oo n— >oo 

This completes the proof. ■ 
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